Skip to content

bidirectional streaming for output filter chains#834

Open
adilhafeez wants to merge 1 commit intomainfrom
adil/streaming-output-filters
Open

bidirectional streaming for output filter chains#834
adilhafeez wants to merge 1 commit intomainfrom
adil/streaming-output-filters

Conversation

@adilhafeez
Copy link
Copy Markdown
Contributor

Summary

  • Replace per-chunk HTTP requests to output filters with a single bidirectional streaming connection per filter, eliminating 50-200+ round-trips per streaming LLM response
  • Filters opt in via streaming: true in config — backward compatible, existing filters unaffected
  • Falls back to per-chunk mode automatically if streaming connection fails or any filter doesn't support it
  • Updates PII deanonymizer demo as reference implementation with request.stream() + StreamingResponse

Made with Cursor

Replace per-chunk HTTP requests to output filters with a single
bidirectional streaming connection per filter. This eliminates
the 50-200+ round-trips per streaming LLM response.

Filters opt in via streaming: true in config. When all output filters
support streaming, brightstaff opens one POST per filter with a streaming
request body (Body::wrap_stream) and reads the streaming response. Filters
that don't opt in fall back to the existing per-chunk behavior.

Updates the PII deanonymizer demo as the reference implementation with
request.stream() + StreamingResponse support.

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant